Open Source Data Science: An R Perspective
author: Joseph B. Rickert date: September 18, 2019 autosize: true
Center for Strategic and Budgetary Assessments
Implications of Data Science as a Resource
Workshop 3
RStudio
Privately held company (Boston)
56 % of resources devoted to building open source and free software for R
Professional products: RStudio Server Pro, Connect and Package Manager
Mission
To provide open source and enterprise-ready professional software for the R statistical computing environment
These tools further the cause of equipping everyone, regardless of means, to participate in a global economy that increasingly rewards data literacy.
R Consortium
left: 45%
Non Profit Membership Corporation - Organized under the Linux Foundation - Governed by a Board of Directors - Technical committee (ISC) funds projects and oversees work - More that $1M awarded so far
Mission
- Support the R Foundation and organizations developing and using R
- Develop and fund Infrastructure
Open Source
Photo by Alex Holyoake
What is Open Source?
Open source software is software with source code that anyone can inspect, modify, and enhance
Open source projects, products, or initiatives embrace and celebrate principles of open exchange, collaborative participation, rapid prototyping, transparency, meritocracy, and community-oriented development
Source: Linux Foundation
For more open source information TODO
Some Open Source Statistics
Open Source Strengths
Open Source has reached “critical mass”: there is widespread interoperability among conceptually related tools: e.g. you can use R or Python (or R and Python) to build deep learning TensorFlow models, run them on Apache Spark clusters, and manage them with Dockercontainers and Kubernetes.
Continuous Validation: The most important and useful open source projects are continuously monitored and tested by thousands of experts worldwide.
Open Source Weaknesses
- Security:
- by definition everyone has access to the source code
- security is not generally a design requirement
- Licenses may impose restrictions
- GPL requires software derived from GPL code and released to the public to also be GPL
- See Package licensing and enterprise use
- Version control and package management require constant attention
The R Project
left: 40%
- 1976: John Chambers, Bell Labs begin work on S
- 1993: Gentleman & Ihaka, U of Auckland begin work on R
Some History - 1995: R released as open-source
- 1997: R Core Group formed
- 1997: CRAN starts with 12 pkgs
- 2000: R 1.0.0 released
- 2001: Bioconductor Project
- 2003: R Foundation formed
- 2004: First useR! conf - 2009: NY Times article on R
- 2015: The R Consortium
- 2019: CRAN near 15K pkgs
S was conceived as an interface
left: 60%
John Chambers’ famous diagram from May 1976 indicates the intention to design a software interface to call an arbitrary FORTRAN subroutine, ABC, by wrapping it in some simplified calling syntax: XABC( ).
The main idea was to bring the best computational facilities to the people doing the analysis. As John phrased it: “combine serious computational challenges with convenience”
R is more than a language!
- The worlds largest open and free repository repository for statistical knowledge containing reference implementations for almost every important statistical algorithm
- The dominant language for statistical computing
- A vibrant community with a stable ecosystem
- A gateway to other computational resources: TensorFlow (Deep Learning)
“It was always understood that R is meant to build on a base of computational tools. R relies on the ability of functions to communicate with, and exchange objects with other software.”
John Chambers, Extending R, CRC Press (2016)
Some Characteristics of the R Language
R is an interpreted scripting language - Base R has a relatively small footprint - The majority of growth and innovation comes from contributed packages: libraries of functions. - Features such as non-standard evaluation make it a good choice for “Design Specific Languages”
R is a functional, object based language - Everything that exists in R is an object - Everything that happens in R is a function call - Interfaces to other software are part of R
A Brief Example
NOAA-Polar-Ice.nb.html
The R Ecosystem
- The R Language
- The R Foundation (An Austrian Foundation that manages the R - Language)
- The R Core Group (20 or so R Foundation members that are - authorized to make commits to R)
- CRAN: the central repository with more than 14,000 tested packages
- Bioconductor: Project for high-throughput genomic analysis
- The R Community: Developers, users, RUGS, R-Ladies, conferences etc.
- The R Consortium
Open Source Data Science
left: 50%
Image by Jingwen Zheng
- ML Algorithms
- Validation
- Workflows
- Interoperable Software
- Reproducibility
- Multiple Data Sources
- Production Pipelines
- Collaboration
Machine Learning with Caret Package
The caret package (short for Classification And REgression Training) is a set of functions that attempt to streamline the process for creating predictive models.
tidymodels: a new framework for predictive modeling
Validation
Package Level - Test Programs are available for inspection and use - E.g. Almost 10K lines of test code for survival package
Industry/company Critical Collections - E.g. Pharmaceutical Industry R Validation Hub
Data Science Workflow
- The tidyverse is a DSL for Data Science
- All packages share an underlying design philosophy, grammar, and data structures.
Interoperable Software: R / Python / etc.
https://www.tensorflow.org ***
- An open-source library for ML from Google Brain
- Associated with Deep Learning
- Is a general purpose library for numerical computation
- Implemented as a data flow graphs programming language
- Hardware Independent (CPU, GPU, TPU)
TensorFlow: Production Grade ML
Key TensorFlow Concepts
See the Google Brain paper from Abadi et al. (2017) for the details.
- Nodes in a
TensorFlowgraph represent mathematical operations - Data organized as multidimensional data arrays (tensors) that flow along the edges.
- Edges may also indicate control signals that constrain the order of execution.
R Interfaces to TensorFlow
left: 60%
- Keras API: A high-level interface to neural networks
- Estimators API: A high-level interface to common models including regression and classifiers
- Core API: A low-level interface that provides full access to the TensorFlow computational Graph
Reproducibility
R Markdown 
Reproducibly integrate code and text
Use notebooks to weave together narrative text and code from several computer languages (including R, Python, and SQL)to produce formatted output for several document types (HTML, pdf, etc) using multiple languages.
Production Data Sources
left: 40%
Connect to Spark Clusters - Use as backend to dplyr - Filter and aggregate Spark data sets - Use Spark’s MLlib
Production Workflows / Pipelines
- Aspects of enterprise-wide data science include the integration of data sources, access and security issues, version control and the management of models and other artifacts
- RSstudio tools address these issues
Production Deployment with Containers
Kubernetes and Docker Pipeline 
Containers: - Package up everything needed to run App - Reliably deploy Apps on multiple platforms - Distribute Apps across clusters
Communicate and Collaborate
Within teams and between departments
- Shiny makes it easy to build interactive web apps
- Host standalone apps on a webpage
- Embed apps in R Markdown documents
- Build dashboards
Some Shiny Examples
For many more see: - The Shiny Gallery - The Shiny User Showcase - showmeshiny
Thank You
joseph.rickert@rstudio.com @RStudioJoe
This presentation available at: https://github.com/joseph-rickert/OSD-Sept-2019
